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ABSTRACT 



This paper describes the design of a study intended to 
examine the devolution of welfare in four cities. The study will measure the 
economic effects of new policies on individual and potential welfare 
recipients; describe the new programs and policies as they are put in place; 
examine the experiences of a small number of families; look at how the new 
policies and funding mechanisms affect both for-profit institutions and 
nonprofit and public service delivery systems; and assess changes in the 
social and economic vitality of neighborhoods. The study focuses on the 
portion of the impact analysis that uses administrative data. It uses a 
multiple cohort design, in which a group of those receiving welfare or at 
risk of receiving welfare will be followed over time, and their outcomes will 
be compared. The paper describes: a model of the impacts of welfare reform 
(the treatments and predicted effects) ; the proposed design (a multiple 
cohort analysis) ; hypothetical examples of multiple cohort analysis; random 
assignment evaluation as multiple cohort analysis; national variation in 
outcomes across cohorts; selecting cohorts; and data sources. An appendix 
presents econometrics of the multiple cohort design. (SM) 
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Over the past few years, our nation’s social safety net has undergone a period of rapid and 
dramatic change, the third such period in sixty years. The last two upheavals, in the 1930s and 
the 1960s, left enduring legacies — Social Security, Medicare, and Medicaid are among the best- 
known examples — and generated far-reaching changes in American society. On August 22, 
1996, President Clinton signed the historic Temporary Assistance for Needy Families (TANF) 
block grant legislation that heralds another period of major change. The reform embodies an in- 
teresting set of contrasts. On the one hand, it grants the states an extraordinary amount of 
flexibility to design whatever mix of services and benefits they think will reduce dependency and 
provide for the needy. On the other hand, the law is quite prescriptive about work participation 
requirements. Specifically, the 1996 law contains a five-year lifetime limit on receipt of federally 
funded cash assistance; authorizes states to impose shorter time limits at their discretion; requires 
that, by the year 2002, 50 percent of all recipients who have received cash aid for two years work 
at least 30 hours a week in order to continue receiving benefits (only about 14 percent of all re- 
cipients meet this definition currently); denies Food Stamp assistance and, at state discretion, 
cash benefits to all legal immigrants; allows states to withdraw 20 percent of the current state 
share of Aid to Families with Dependent Children (AFDC) costs; and permits 30 percent of all 
federal funds to be diverted to child care and other programs for non-AFDC recipients. 

The Manpower Demonstration Research Corporation (MDRC) has undertaken an initia- 
tive that is examining how states, urban counties, and large cities restructure social welfare pro- 
grams over the next several years. The overarching goal of its Project on Devolution and Urban 
Change is to assess whether “devolution” is generating the dramatic changes in policy and pro- 
grams predicted by supporters and critics; just as important is to understand what difference these 
policies are making in the lives of low-income families. Begun in early 1997, the project will last 
approximately five years. 

Welfare dependency and poverty have become increasingly concentrated in America’s 
largest cities. Furthermore, welfare agencies in large cities usually have had little success in mov- 
ing recipients into jobs, so that the 1996 legislation poses significant implementation challenges 
in these settings. For these reasons, the Urban Change project is taking place in four large urban 
counties: Philadelphia, Los Angeles, Miami-Dade, and Cuyahoga (Ohio), which includes the city 
of Cleveland. 

To provide depth and breadth to the analysis of welfare’s changes, the study includes five 
major components: 

1 . An impact study to measure the economic effects of the new policies on individual wel- 
fare recipients and potential welfare recipients 

2. An implementation study to describe the new programs and policies as they are put in 
place 

3. An ethnographic study to look in depth at the experiences of a small number of families 

4. An institutional study to examine how the new policies and funding mechanisms affect 
both for-profit institutions and nonprofit and public service delivery systems 
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5. A neighborhood indicators study to assess changes in the social and economic vitality of 
neighborhoods 

The purpose of the Urban Change project is twofold: to describe income sources, income 
amounts, and well-being after devolution; and to infer the impacts of devolution on individuals, 
communities, and institutions by estimating what would have happened under AFDC. Although 
the five study components are designed to build on one another, and all five will be used in de- 
termining the effects of recent reform, this design paper focuses on the first component, the indi- 
vidual-level impact analysis. Information for the individual impact component will come from 
two sources. Administrative data on AFDC, TANF, Food Stamps, and earnings will be collected 
for all nonelderly parents of minor children who ever received Food Stamps or AFDC/TANF be- 
tween 1992 and 2002 in the four counties. In addition, 2,000 single mothers receiving welfare in 
high-poverty neighborhoods will be surveyed in each county’s principal city. While the adminis- 
trative data will allow us to precisely estimate changes in welfare and employment over time, the 
surveys will provide crucial information on other sources of income, child and parent well-being, 
family formation, experiences with the welfare system, and attitudes toward welfare and work. 

This paper focuses on the portion of the impact analysis that uses administrative data. To 
infer impacts of the new policy on employment, earnings, and welfare, the study is using a multi- 
ple cohort design. In this design, a cohort, or group, of those receiving welfare or at risk of re- 
ceiving welfare will be followed over time, and their outcomes will be compared. If patterns of 
behavior for cohorts who pass through reform differ markedly from patterns for those who are 
subject to the defunct rules of AFDC, this will be taken as evidence that welfare reform has had 
an impact. The remainder of this paper describes the multiple cohort design, investigates the 
power of the technique using data from a variety of sources, and discusses analytical issues 
which remain to be addressed. 

I. A Model of the Impacts of Welfare Reform: The Treatments 
and Predicted Effects 

The likely impacts of welfare reform on state welfare agencies and on the working and 
welfare poor are unclear. Figure 1 provides a conceptual framework for understanding the proc- 
ess by which welfare reform is expected to affect individual welfare recipients, potential welfare 
recipients, and their families and children. The two boxes and two circles at the top of the figure 
describe the primary determinants of individual outcomes and changes in individual behavior. 
The middle boxes capture the two primary components of the welfare reform bill: (1) the TANF 
legislation, including the specific rules and regulations embedded in it, and (2) the devolution to 
the states of responsibility for design and control of cash welfare programs. 

These two welfare reform boxes are separated in the flowchart to underscore the ambigu- 
ity inherent in the welfare reform legislation, which constitutes the principal reason why its im- 
pacts are difficult to predict. First, the legislation was billed as an effort to curtail and regulate 
welfare receipt. The provisions regarding time limits and work requirements are consistent with 
this emphasis. However, the legislation also transferred power and discretion from the federal 
government to the states. In practice, increased state discretion sometimes offsets the effects of 
the more restrictive parts of the legislation. For example, some states have developed programs 
to extend benefits to those who become ineligible under strict TANF rules. 
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Figure 1 

A Model of the Impacts of Welfare Reform on Individuals 
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To the sides of the two welfare reform boxes at the top of Figure 1 are two circles depict- 
ing external factors: the economic context and individual characteristics. Changes in these exter- 
nal forces affect not only what the ultimate outcomes will look like (as indicated by the bold 
dashed line running across and then down the left side of the figure) but presumably also what 
states will do with their newfound freedom to design welfare programs. 

The next steps in the flowchart are the design and implementation of state welfare pro- 
grams reflecting TANF rules, state policymaking, and economic constraints. As with the TANF 
legislation, it cannot be assumed that whatever is written down in state plans and state law ade- 
quately defines the new welfare environment in the cities under study. In the model, a diamond 
(local implementation) separates state legislation from the actual “treatments” affecting current 
and future welfare recipients. This highlights the importance of the implementation study both 
for the Urban Change project generally and for the impact analysis in particular. If the implemen- 
tation study indicates that welfare reform is not associated with real changes at the level of local 
welfare offices, then the impact analysis cannot be expected to detect much about the effects of 
the new legislation. 

The arrows lead from the implementation diamond at the center of Figure 1 to the heart of 
the impact evaluation of welfare reform: the definition of the welfare reform treatments, which 
include the actual changes in eligibility rules, benefits, and obligations that will directly affect 
people’s outcomes and behavior. The model shows six of the key changes that have been imple- 
mented, namely, (1) limits on eligibility to receive cash welfare, (2) increased sanctioning to in- 
duce welfare recipients to find work or participate in required activities and increased hassle to 
discourage individuals from applying for welfare, (3) increased earnings disregards to stimulate 
employment, (4) time limits, (5) increased resources for child care to help welfare recipients pay 
for one of the key work-related expenses, and (6) increased resources for job placement and 
training services to help welfare recipients find jobs and gain skills needed for employment. To- 
gether with the individual characteristics of the people affected and the economic context in 
which they live, these six treatment changes are expected to shape changes in outcomes and indi- 
vidual behavior. As discussed below, the key challenge in developing a viable research design is 
to find one that allows the effects of welfare reform to be isolated from demographic changes in 
the caseload and from changes in the local economy. 

The bottom part of Figure 1 lists some outcomes associated with welfare reform and di- 
vides them into two levels. The first level includes outcomes on which welfare reform should 
have more direct or immediate effects. These outcomes, which can be measured using large sam- 
ples from administrative data, include caseloads, welfare grants, employment levels among wel- 
fare recipients, and lengths of stay on welfare. The second level of outcomes is affected less di- 
rectly and will be measured using smaller samples from survey data. These include family in- 
come, housing status, and other measures of child and family well-being. 

The legislation passed in the four states of the study includes several major changes in 
welfare (see Table 1). Perhaps the most publicized change is a limit on the number of months 
that a parent can receive welfare. The federal legislation limits lifetime receipt of TANF benefits 
to 60 months, although each state is allowed to exempt 20 percent of cases from this lifetime 
limit. Of our four states, only Florida has set a lower limit on lifetime receipt, 48 months. In addi- 
tion to the lifetime limit, three of the four states in the Urban Change project limit the number of 
consecutive months that welfare can be received. In California, a current recipient must leave 
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SOURCES: ‘ “What Welfare Reform Will Mean to the People of the State of Ohio” (Ohio Department of Human Services, July 1997). 
b ‘Welfare Reform Becomes a Reality” (State of California Web site, January 1997). 

c “California Welfare Reform: Summary of the Provisions of AB 1542 and AB 1008” (County Welfare Directors Association of California, August 1997). 
d “A Plan to Reform Welfare in Florida WAGES: Work and Gain Economic Self-Sufficiency” (Florida State Legislature, May 1996). 
e “TANF State Plan” (Pennsylvania Department of Public Welfare, enacted March 1997). 



the welfare rolls for at least one month after receiving benefits for 24 consecutive months; an ap- 
plicant must leave the rolls after receiving benefits for 18 consecutive months. In Ohio, welfare 
can be received for 36 months in any five-year period. In Florida, a current recipient can receive 
welfare for 36 months in a six-year period, while a new applicant is restricted to 24 months of 
receipt in a five-year period. 

The second major change in policy affects earnings disregards, allowing welfare recipi- 
ents to keep more of their earnings. Under the rules of AFDC, a working welfare recipient was 
allowed to keep the first $120 of monthly earnings plus one-third of remaining earnings for the 
first four months of employment. For the next eight months, the working welfare recipient could 
keep $120 of earnings per month before having her grant reduced. After a year of working, the 
recipient could keep only $90 of earnings. Under the new rules in Florida, California, and Ohio, ./ 
the fixed portion of the earnings disregard has been increased to $200 or more. In all four states, 
a TANF recipient can keep one-half of remaining earnings after the fixed disregard. In addition, 
in Florida, California, and Pennsylvania, the enhanced disregard is available no matter how long 
the welfare recipient combines work and welfare; in Ohio, it is available for 18 months. 

The third major change in policy requires recipients to engage in work activities. One 
parent in each two-parent household must be in work activities for at least 35 hours per week. 
For single-parent cases, hours of required work were 20 hours per week in fiscal year 1997 and 
are 30 hours per week in fiscal year 2000. States must gradually increase the percentage of re- 
cipients meeting this work requirement; 25 percent of single-parent recipients were supposed to 
meet the work requirement in fiscal year 1997, but 50 percent are supposed to meet it in fiscal 
year 2002; of two-parent cases, 75 percent were supposed to meet the work requirement in fiscal 
year 1997, but 90 percent are supposed to meet it in fiscal year 1999 and later. 

In addition to these changes in major treatments, welfare rules have changed in other 
ways. In Philadelphia, recipients are allowed to own one car of unlimited value, whereas Flor- 
ida’s recipients are allowed to own a car worth no more than $8,500. In Ohio, Florida, and Penn- 
t sylvania, the $50 child support pass-through was eliminated. In California and Florida, a family 

cap limits the increase in benefits for children added to a family after the family starts receiving 
TANF benefits. Some states will provide additional transitional benefits to encourage recipients 
to leave welfare for work. 

The combined effects of these changes is less than clear. Consider the effects of the en- 
hanced earnings disregards and increased child care assistance. Both components of welfare re- 
form make work more attractive to current welfare recipients. Among recipients who are not cur- 
rently working, this should lead to increased hours of work. However, enhanced disregards will 
encourage work only among people who can remain eligible for welfare while working. For this 
group, then, the two policies will increase work by a limited amount but will not reduce welfare 
use immediately. 

Among recipients who already work, the effects of earnings disregards and child care as- 
sistance on hours worked are ambiguous. Both policy changes allow recipients to keep more of 
each additional dollar earned, and this provides an incentive to work more. Because they also al- 
low an individual to maintain the same material living standard and to work less, however, they 
could cause hours of work to decline. In any case, because welfare is more compatible with work, 
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these welfare recipients are expected to stay on welfare longer. Such behavior would have the 
effect of increasing the caseload. 

For people not currently receiving welfare, earnings disregards and child care assistance 
make welfare more attractive. In the absence of some countervailing condition that makes wel- 
fare unattractive, this group’s welfare use should increase. Further, because of the availability of 
welfare, their hours worked should decline. 

Because of the combined effects of earnings disregards and child care assistance for the 
different groups, it is clear that the welfare caseload is likely to increase and that employment is 
likely to increase, but hours worked could either rise or fall. Total hours worked could decline 
even among recipients, although that is very unlikely. Over the longer term, the effects of these 
policies are even less clear. Although enhanced disregards and child care assistance will initially 
increase welfare caseloads, the work experience acquired could lead to higher-paying employ- 
ment opportunities, which would entail both greater work effort and a lower caseload. 

The presence of time limits does not alter the foregoing analysis or diminish the ambigu- 
ity associated with these components of welfare reform. Rather, it increases the uncertainty about 
the overall effects of welfare reform on some outcomes. For some individuals, time limits might 
make welfare less valuable and thus discourage welfare use. For these individuals, hours worked 
are expected to increase. For many potential recipients, however, the time limits do not constitute 
a constraint on behavior, and policies such as enhanced disregards make welfare a more attrac- 
tive form of temporary assistance, increasing welfare use and perhaps lowering hours worked. 
Overall, then, time limits should increase the effects of welfare reform on employment, make the 
effects on welfare caseloads unclear, and leave the effects on hours worked ambiguous. 

The ambiguities associated with the effects of the likely components of welfare reform 
motivates the Urban Change impact study. Because theory provides little guidance about the im- 
pacts of reform, only empirical analysis can shed light on the study’s key questions. Both positive 
and negative impacts may occur, leaving open the question of the magnitude of these effects — 
and the net impact of TANF on individuals, their communities, and public sector budgets. 

II. The Proposed Design: A Multiple Cohort Analysis 

The process of measuring the effects of welfare reform on individual outcomes has three 
distinct steps. The first step is to measure the outcomes of interest for those affected by the 
changing welfare environment. The second step is to identify a counterfactual state for the chang- 
ing welfare environment and to measure outcomes for this counterfactual state. The third step is 
to compare outcomes for the changed environment and its counterfactual, thereby calculating the 
impact of the change. Expressed as a simple equation, this looks as follows: 

Impact Welfare Change = (OutcomelWelfare Change) - (OutcomeINo Welfare Change) 

The last term in this relation is the counterfactual. Because it is easy to measure outcomes for 
those affected by the policy change, the search for an appropriate research design centers on find- 
ing the best method of estimating the counterfactual. 
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Alternative I: Random Assignment 



The research design most commonly used by MDRC to obtain the counterfactual out- 
come is random assignment of some individuals to a control group that is denied access to the 
program being studied. As is widely understood, the randomness of the procedure by which a 
control group is created protects the estimated program impacts from virtually all potential 
sources of bias. However, random assignment has several important drawbacks. First, it is ex- 
pensive and difficult to implement. Second, it often raises ethical questions. The third and most 
important drawback is that random assignment is useful only in situations in which the control 
group represents the intended counterfactual. Arguably, the changes in U.S. welfare policy since 
1996 do not constitute such a situation. Although it is theoretically possible to randomly assign 
welfare recipients to a traditional AFDC program — kept in place for research purposes only — 
it would be difficult to isolate traditional AFDC control group members from the system-wide 
and community-wide changes going on around them. Therefore, a random assignment design 
does not seem a feasible option for this evaluation of welfare reform. 

Alternative II: Cross-Sectional Comparisons 

A related nonexperimental method of evaluating a new policy is to compare outcomes for 
different individuals, some of whom are subject to welfare reform, some of whom are not. This 
type of approach requires constructing or finding a comparison group that is the nonexperimental 
equivalent of the control group used in experimental evaluations. For a comparison group to pro- 
vide an appropriate counterfactual, two conditions must be met, namely, (1) that the comparison 
group not be subject to a treatment and (2) that the comparison group be comparable to the pro- 
gram group in all aspects relevant to both the treatment and the outcome. 

In evaluating the effects of TANF, neither requirement can be met. The global nature of 
the welfare changes makes it unlikely that a relevant comparison group can be found that is not 
itself affected by welfare reform. Nonexperimental evaluations of other policy changes have 
sometimes used variation in programs across states or counties to isolate a program’s effects. 
However, all states were affected by the TANF legislation, and even though there is variation in 
welfare programs across the states, such variation is probably insufficient to produce a compel- 
ling counterfactual for welfare reform itself. 

Alternative III: Time-Series Analysis 

Many policy changes have been evaluated by examining changes in outcomes over time. 
There are many different types of such time-series designs, but ultimately they all use the past as 
the primary source from which to create a counterfactual to measure the effects of an interven- 
tion. Time-series analyses are popular, because they are intuitively appealing. They also offer a 
very compelling source of causal inference by imposing a temporal order on otherwise ambigu- 
ous relationships among variables. 

The simplest example of a time-series design is known as a before-after analysis, which 
is easy to understand and quite popular with journalists, politicians, and voters. This design sim- 
ply compares an outcome measured before a change takes effect with the same outcome meas- 
ured after the change. A hypothetical example of a before-after analysis would be a comparison 
of Los Angeles County’s welfare caseloads in 1994 and 2000. Such a design would not be con- 
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sidered for the Urban Change project because of the many problems inherent in such a compari- 
son. These problems manifest themselves as three different threats to the internal validity of the 
analysis, namely, (1) maturation, (2) history, and (3) regression to the mean (Cook and Campbell, 
1979). Following is a brief discussion of these validity threats. 

Maturation. Maturation bias is the most obvious threat to validity and also the easiest to 
eliminate. Maturation represents the idea that outcomes may gradually change over time as a re- 
sult of natural processes. For example, any welfare recipient will, over time, leave the rolls as her 
youngest child reaches the age of 18. Or, in another example, a city’s rate of welfare receipt may 
gradually increase as middle-class households move away to suburbs outside the city limits. 

Maturation bias in a two-point time-series design is illustrated in Figure 2, in which time 
is represented on the horizontal axis while a generic outcome Y is represented on the vertical 
axis. The researcher observes outcomes only at two points — point A, showing the outcome prior 
to the policy change, and point B, showing the outcome after the policy change. The dotted line 
shows an unobserved underlying trend in Y. Without knowing the trend, a researcher might at- 
tribute the change in Y between points A and B to the program, estimating its effect on Y to be b- 
& However, the trend shows that this conclusion is wrong. Pre-program measure A is not a good 
counterfactual for post-program measure B. Instead, the appropriate comparison would be be- 
tween points C and B, producing an estimated effect of b-c, which is the exact opposite of the 
original estimate. To address this problem, the researcher needs to add data about point D to the 
analysis, thereby allowing one to predict the location of point C (by extrapolating the line DA) 
and forming an appropriate counterfactual for post-program outcome B. This means that, given a 
stable trend in Y, all that is needed to prevent maturation bias is to find one or more pre-program 
points D to obtain a good estimate of the counterfactual C. 

History. A more difficult problem is created by the second validity threat, history. In 
studying the effects of a policy change, the problem of history occurs when a second, unrelated 
change occurs after the new policy goes into effect. Figure 3 provides an example. 

Assume that there are sufficient pre-program data to estimate a trend for outcome Y. That 
is, assume that the researcher can observe point D and point A and, by extrapolation of line DA, 
that point C is chosen as the counterfactual. To this researcher, b-c appears to be the impact of 
the new program. Unknown to the researcher, however, a second event has occurred that affects 
outcome Y. An example of this event could be the enhancement of the local JOBS (Job Opportu- 
nities and Basic Skills Training) program or a state-mandated reduction in welfare grants, unre- 
lated to the new TANF legislation. In the example shown in Figure 3, the event changed the di- 
rection of the underlying (unobserved) trend line, directing it downward, toward point C'. This 
means that the true impact of the second event (welfare reform) is b-c' instead of b-c. The differ- 
ence is attributable to history bias. 

The extent of history problems in time-series analyses is determined by a number of dif- 
ferent factors, including (1) the number and spacing of individual observations of Y, (2) the abil- 
ity to identify the event causing the bias, (3) the ability to separate the timing of different events, 
and (4) the size of the spurious effect on Y relative to the program effect. In other words, a time- 
series design is least likely to suffer from history bias if there are observations at many points in 
time, if important events are widely spaced, if there are data for all relevant time-varying vari- 
ables, if changes are sudden and timing is precise, and if program effects are large compared 
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with anything else that may occur simultaneously. (The discussion below about more developed 
research designs will return to these criteria.) 

Regression to the mean. This third type of bias prevalent in time-series designs is a spe- 
cial type of maturation bias that is much more difficult to control with pre-program data because 
the underlying trend is not linear. Consider Figure 4. The dotted line shown in earlier figures has 
been replaced by a slow-moving “wave” that is centered on an underlying trend line (not shown). 
From each extreme point of the wave, the outcome has a tendency to move back toward the un- 
derlying mean. This movement reflects a “natural” tendency and is considered unrelated to the 
underlying trend and also unrelated to any events happening along the way. 

In many situations, regression to the mean is a simple source of random variation, which 
does not cause estimates to be systematically biased. However, there are situations in which the 
selection of a sample or its exposure to a treatment systematically occurs at the same time as an 
extreme in the oscillating trend. Applying to receive welfare is a good example of such a situa- 
tion. At the time of application, people are likely to have reached a natural low in their earnings 
and other resources; the implication is that those outcomes will likely increase by themselves 
without any additional intervention. Such a low is often referred to as the “pre-program dip,” and 
it is a serious problem in many evaluations that rely on time-series designs. 

Figure 4 shows an extreme example of the potential bias from regression to the mean. Us- 
ing points D and A to estimate the underlying trend, a counterfactual is created at point C. How- 
ever, the real trend follows the dotted line (rather than the dashed one), leading to the real unob- 
served counterfactual at point C'. The bias is the sum of be' and cb. 

The only way to prevent a bias such as this one (while still remaining within a time-series 
framework) is to collect enough pre-program data to reliably estimate the underlying trend line. If 
the periodic ups and downs in Y are spaced widely apart, however, this means going back in time 
several years at least. Such history may not always be available. 1 

Alternative IV: Combinations of Cross-Sections Over Time (Cohort Analysis) 

An approach that combines the advantages of cross-sectional analysis with the advantages 
of time-series analysis would use information about several cross-sections chosen at different 
points in time. An analysis using this type of approach is commonly referred to as cohort analy- 
sis, because it compares different cohorts of people drawn from essentially the same population 
over time. Cohort analyses have been popular in situations where there is frequent turnover of the 
population of interest, for example, in the evaluation of school reform programs, where the same 
grade can be compared over multiple school years. 

In a cohort analysis, some of the cross-sections are selected from data collected before 
welfare reform, and some are selected from data collected after welfare reform. The cohorts se- 
lected before welfare reform provide comparison groups that have the advantage of not having 



'Note that on an aggregate level, regression to the mean can also be a serious problem. As particular social prob- 
lems (such as crime) reach periodic peaks, new policies to combat them may gain support and be put into place. The 
social problem subsequently declines, leaving policymakers and politicians proud of their achievement. Next, inter- 
est in the social problem weakens, funding goes down, and the social problem begins a new (natural) upswing, gen- 
erating calls for increased funding and new programs once again. 
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